Information Theoretic Multi-modal Signal Processing

We present an information theoretic approach to multi-modal signal processing, and validate the framework with illustrative examples from medical image processing and multi-media signal processing. The approach tries to adaptively identify that pair of feature space representations of a pair of multi-modal signals which carries maximal redundancy. Analogously, we can say that the framework extracts simultaneously the features of both signals of a multi-modal signal pair where the extraction criteria is maximum redundancy between the resulting feature space representations. The framework is based on stochastic processes, Markov chains, and error probabilities. Extracting such feature pairs of multi-modal signals has a wide range of potential applications, some of which will be used to illustrate the framework with practical results. In particular, we will show how our approach can efficiently be used for multi-modal medical image processing and for joint analysis of multi-media sequences (audio and video).